Masked image modeling (MIM), an emerging self-supervised pre-training method,
has shown impressive success across numerous downstream vision tasks with
Vision transformers (ViTs). Its underlying idea is simple: a portion of the
input image is randomly masked out and then reconstructed via the pre-text
task. However, the working principle behind MIM is not well explained, and
previous studies insist that MIM primarily works for the Transformer family but
is incompatible with CNNs. In this paper, we first study interactions among
patches to understand what knowledge is learned and how it is acquired via the
MIM task. We observe that MIM essentially teaches the model to learn better
middle-order interactions among patches and extract more generalized features.
Based on this fact, we propose an Architecture-Agnostic Masked Image Modeling
framework (A$^2$MIM), which is compatible with both Transformers and CNNs in a
unified way. Extensive experiments on popular benchmarks show that our A$^2$MIM
learns better representations without explicit design and endows the backbone
model with the stronger capability to transfer to various downstream tasks for
both Transformers and CNNs.

本文提出了一种基于蒙版图像建模的框架，即 A^2MIM，可用于 Transformers 和 CNNs 网络，通过对补丁之间的相互作用的研究发现蒙版图像建模实质上教授了模型更好地处理中阶交互和提取通用特征的能力，并通过大量实验证明了该方法在不需要显式设计的情况下学习到更好的表示，并赋予骨干模型更强的能力，以适应于不同的下游任务。